Co-authored-by: chen, suyue <[email protected]>

…t#568)

* fix mismatched response format w/wo streaming  guardrails

* fix & debug

* fix & rm debug

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Dong, Bo1 <[email protected]>

Co-authored-by: chen, suyue <[email protected]>

…t#568)

* fix mismatched response format w/wo streaming  guardrails

* fix & debug

* fix & rm debug

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* add rerank with neural speed

Signed-off-by: Dong, Bo1 <[email protected]>

* add the code

Signed-off-by: Dong, Bo1 <[email protected]>

* add the code

Signed-off-by: Dong, Bo1 <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: Dong, Bo1 <[email protected]>

* fix mismatched response format w/wo streaming guardrails (#568)

* fix mismatched response format w/wo streaming  guardrails

* fix & debug

* fix & rm debug

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Dong, Bo1 <[email protected]>

* Fix guardrails out handle logics for space linebreak and quote (#571)

* fix mismatched response format w/wo streaming  guardrails

* fix & debug

* fix & rm debug

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* debug

* debug

* debug

* fix pre-space and linebreak

* fix pre-space and linebreak

* fix single/double quote

* fix single/double quote

* remove debug

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Dong, Bo1 <[email protected]>

* BUG FIX: LVM security fix (#572)

* add url validator

Signed-off-by: BaoHuiling <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add validation for video_url

Signed-off-by: BaoHuiling <[email protected]>

---------

Signed-off-by: BaoHuiling <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Dong, Bo1 <[email protected]>

* Modify output messages. (#569)

* Reduced output.

Signed-off-by: zepan <[email protected]>

* Output the location where the modified Dockerfile file is referenced.

Signed-off-by: zepan <[email protected]>

* for test

Signed-off-by: zepan <[email protected]>

* Restore test file.

Signed-off-by: zepan <[email protected]>

---------

Signed-off-by: zepan <[email protected]>
Signed-off-by: Dong, Bo1 <[email protected]>

* refine logging code. (#559)

* add ut and refine logging code.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update microservice port.

---------

Co-authored-by: root <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Dong, Bo1 <[email protected]>

* adding lancedb to langchain vectorstores (#291)

* adding lancedb to langchain vectorstores

Signed-off-by: sharanshirodkar7 <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: sharanshirodkar7 <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: lvliang-intel <[email protected]>
Signed-off-by: Dong, Bo1 <[email protected]>

* Refine Dataprep Milvus MS (#570)

Signed-off-by: letonghan <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Dong, Bo1 <[email protected]>

* final version

Signed-off-by: Dong, Bo1 <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: Dong, Bo1 <[email protected]>

* update the readme

Signed-off-by: Dong, Bo1 <[email protected]>

* add the sign

Signed-off-by: Dong, Bo1 <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: Dong, Bo1 <[email protected]>

* fix error for pre ci

Signed-off-by: Dong, Bo1 <[email protected]>

* add the ut

Signed-off-by: Dong, Bo1 <[email protected]>

* update docker file

Signed-off-by: Dong, Bo1 <[email protected]>

* update CI test log achieve (#577)

Signed-off-by: chensuyue <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Dong, Bo1 <[email protected]>

* Multimodal dataprep (#575)

* multimodal embedding for MM RAG for videos

Signed-off-by: Tiep Le <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* develop data prep first commit

Signed-off-by: Tiep Le <[email protected]>

* develop dataprep microservice for multimodal data

Signed-off-by: Tiep Le <[email protected]>

* multimodal langchain for dataprep

Signed-off-by: Tiep Le <[email protected]>

* update README

Signed-off-by: Tiep Le <[email protected]>

* update README

Signed-off-by: Tiep Le <[email protected]>

* update README

Signed-off-by: Tiep Le <[email protected]>

* update README

Signed-off-by: Tiep Le <[email protected]>

* cosmetic

Signed-off-by: Tiep Le <[email protected]>

* test for multimodal dataprep

Signed-off-by: Tiep Le <[email protected]>

* update test

Signed-off-by: Tiep Le <[email protected]>

* update test

Signed-off-by: Tiep Le <[email protected]>

* update test

Signed-off-by: Tiep Le <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* cosmetic update

Signed-off-by: Tiep Le <[email protected]>

* remove langsmith

Signed-off-by: Tiep Le <[email protected]>

* update API to remove /dataprep from API names and remove langsmith

Signed-off-by: Tiep Le <[email protected]>

* update test

Signed-off-by: Tiep Le <[email protected]>

* update the error message per PR reviewer

Signed-off-by: Tiep Le <[email protected]>

---------

Signed-off-by: Tiep Le <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Dong, Bo1 <[email protected]>

* add: Pathway vector store and retriever as LangChain component (#342)

* nb

Signed-off-by: Berke <[email protected]>

* init changes

Signed-off-by: Berke <[email protected]>

* docker

Signed-off-by: Berke <[email protected]>

* example data

Signed-off-by: Berke <[email protected]>

* docs(readme): update, add commands

Signed-off-by: Berke <[email protected]>

* fix: formatting, data sources

Signed-off-by: Berke <[email protected]>

* docs(readme): update instructions, add comments

Signed-off-by: Berke <[email protected]>

* fix: rm unused parts

Signed-off-by: Berke <[email protected]>

* fix: image name, compose env vars

Signed-off-by: Berke <[email protected]>

* fix: rm unused part

Signed-off-by: Berke <[email protected]>

* fix: logging name

Signed-off-by: Berke <[email protected]>

* fix: env var

Signed-off-by: Berke <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: Berke <[email protected]>

* fix: rename pw docker

Signed-off-by: Berke <[email protected]>

* docs(readme): update input sources

Signed-off-by: Berke <[email protected]>

* nb

Signed-off-by: Berke <[email protected]>

* init changes

Signed-off-by: Berke <[email protected]>

* fix: formatting, data sources

Signed-off-by: Berke <[email protected]>

* docs(readme): update instructions, add comments

Signed-off-by: Berke <[email protected]>

* fix: rm unused part

Signed-off-by: Berke <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: Berke <[email protected]>

* fix: rename pw docker

Signed-off-by: Berke <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: Berke <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* feat: mv vector store, naming, clarify instructions, improve ingestion components

Signed-off-by: Berke <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* tests: add pw retriever test
fix: update docker to include libmagic

Signed-off-by: Berke <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* implement suggestions from review, entrypoint, reqs, comments, https_proxy.

Signed-off-by: Berke <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix: update docker tags in test and readme

Signed-off-by: Berke <[email protected]>

* tests: add separate pathway vectorstore test

Signed-off-by: Berke <[email protected]>

---------

Signed-off-by: Berke <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Sihan Chen <[email protected]>
Signed-off-by: Dong, Bo1 <[email protected]>

* Add local Rerank microservice for VideoRAGQnA (#496)

* initial commit

Signed-off-by: BaoHuiling <[email protected]>

* save

Signed-off-by: BaoHuiling <[email protected]>

* add readme, test script, fix bug

Signed-off-by: BaoHuiling <[email protected]>

* update video URL

Signed-off-by: BaoHuiling <[email protected]>

* use default

Signed-off-by: BaoHuiling <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update core dependency

Signed-off-by: BaoHuiling <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* use p 5000

Signed-off-by: BaoHuiling <[email protected]>

* use 5037

Signed-off-by: BaoHuiling <[email protected]>

* update ctnr name

Signed-off-by: BaoHuiling <[email protected]>

* remove langsmith

Signed-off-by: BaoHuiling <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add rerank algo desc in readme

Signed-off-by: BaoHuiling <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: BaoHuiling <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: chen, suyue <[email protected]>
Signed-off-by: Dong, Bo1 <[email protected]>

* Add Scan Container. (#560)

Signed-off-by: zepan <[email protected]>
Signed-off-by: Dong, Bo1 <[email protected]>

* fix SearchedMultimodalDoc in docarray (#583)

Signed-off-by: BaoHuiling <[email protected]>
Signed-off-by: Dong, Bo1 <[email protected]>

* update image build yaml (#529)

Signed-off-by: chensuyue <[email protected]>
Signed-off-by: zepan <[email protected]>
Signed-off-by: Dong, Bo1 <[email protected]>

* add microservice for intent detection (#131)

* add microservice for intent detection

Signed-off-by: Liangyx2 <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update license copyright

Signed-off-by: Liangyx2 <[email protected]>

* add ut

Signed-off-by: Liangyx2 <[email protected]>

* refine

Signed-off-by: Liangyx2 <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update folder

Signed-off-by: Liangyx2 <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix test

Signed-off-by: Liangyx2 <[email protected]>

---------

Signed-off-by: Liangyx2 <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Dong, Bo1 <[email protected]>

* Make the scanning method optional. (#580)

Signed-off-by: zepan <[email protected]>
Signed-off-by: Dong, Bo1 <[email protected]>

* add code owners (#586)

Signed-off-by: Dong, Bo1 <[email protected]>

* remove revision for tei (#584)

Signed-off-by: letonghan <[email protected]>
Signed-off-by: Dong, Bo1 <[email protected]>

* Bug fix (#591)

* Check if the document exists.

Signed-off-by: zepan <[email protected]>

* Add flag output.

Signed-off-by: zepan <[email protected]>

* Modify nginx readme.

Signed-off-by: zepan <[email protected]>

* Modify document detection logic

Signed-off-by: zepan <[email protected]>

---------

Signed-off-by: zepan <[email protected]>
Signed-off-by: Dong, Bo1 <[email protected]>

* fix ut issue

Signed-off-by: Dong, Bo1 <[email protected]>

* merge the main

Signed-off-by: Dong, Bo1 <[email protected]>

* align with new pipeline

Signed-off-by: Dong, Bo1 <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* align with newest pipeline

Signed-off-by: Dong, Bo1 <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* upload code

Signed-off-by: Dong, Bo1 <[email protected]>

* update the ut

Signed-off-by: Dong, Bo1 <[email protected]>

* add docker path

Signed-off-by: Dong, Bo1 <[email protected]>

* add the docker path

Signed-off-by: Dong, Bo1 <[email protected]>

---------

Signed-off-by: Dong, Bo1 <[email protected]>
Signed-off-by: BaoHuiling <[email protected]>
Signed-off-by: zepan <[email protected]>
Signed-off-by: sharanshirodkar7 <[email protected]>
Signed-off-by: letonghan <[email protected]>
Signed-off-by: chensuyue <[email protected]>
Signed-off-by: Tiep Le <[email protected]>
Signed-off-by: Berke <[email protected]>
Signed-off-by: Liangyx2 <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Sihan Chen <[email protected]>
Co-authored-by: Huiling Bao <[email protected]>
Co-authored-by: ZePan110 <[email protected]>
Co-authored-by: lkk <[email protected]>
Co-authored-by: root <[email protected]>
Co-authored-by: Sharan Shirodkar <[email protected]>
Co-authored-by: lvliang-intel <[email protected]>
Co-authored-by: Letong Han <[email protected]>
Co-authored-by: chen, suyue <[email protected]>
Co-authored-by: Tiep Le <[email protected]>
Co-authored-by: berkecanrizai <[email protected]>
Co-authored-by: Liangyx2 <[email protected]>
Co-authored-by: kevinintel <[email protected]>